Less Is More: Pay Less Attention in Vision Transformers

Transformers have become one of the dominant architectures in deep learning, particularly as a powerful alternative to convolutional neural networks (CNNs) computer vision. However, Transformer training and inference previous works can be prohibitively expensive due quadratic complexity self-attention over long sequence representations, especially for high-resolution dense prediction tasks. To this end, we present novel Less attention vIsion (LIT), building upon fact that early layers still focus on local patterns bring minor benefits recent hierarchical vision Transformers. Specifically, propose where use pure multi-layer perceptrons (MLPs) encode rich stages while applying modules capture longer dependencies deeper layers. Moreover, further learned deformable token merging module adaptively fuse informative patches non-uniform manner. The proposed LIT achieves promising performance image recognition tasks, including classification, object detection instance segmentation, serving strong backbone many Code is available at https://github.com/zip-group/LIT.